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WHAT IS CLAIMED IS: 



1 1. A machine-implemented image processing method, comprising: 

2 computing a respective motion map for each pairing of a reference image 

3 and a respective image neighboring the reference image in a sequence of base 

4 images, each motion map comprising a set of motion vectors mapping reference 

5 image pixels to respective neighboring image pixels; 

6 assigning respective regions of a target image to motion classes based on 

7 the computed motion maps, the target image having a target resolution level and 

8 the base images having a base resolution level equal to or lower than the target 

9 resolution level; and 

10 computing pixel values for the target image based on corresponding pixel 

1 1 value contributions from the base images selected in accordance with the motion 

12 classes assigned to the target image regions. 

1 2. The method of claim 1, wherein computing motion maps comprises 

2 generating for each image pair respective dense motion vectors describing motion 

3 at pixel locations with respective sets of parameters in a motion parameter space 

4 at sub-pixel accuracy. 

1 3. The method of claim 1, wherein assigning regions of the target 

2 image to motion classes comprises assigning pixels of the reference image to 

3 respective motion classes and up-projecting the motion class assignments to 

4 pixels of the target image. 

1 4. The method of claim 3, wherein assigning reference image pixels to 

2 motion classes comprises computing motion magnitude maps from each motion 

3 map, down-sampling the computed motion magnitude maps to a pyramid of 

4 motion magnitude maps at respective resolution levels lower than the base 

5 resolution level, and segmenting pixels in the pyramid of down-sampled motion 

6 magnitude maps into motion classes. 

7 5. The method of claim 4, wherein assigning reference image pixels to 

8 motion classes further comprises iteratively segmenting pixels in the pyramid of 

9 motion magnitude maps from a coarsest resolution level up to the base resolution 
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10 level, wherein segmentation from each coarser resolution level is up-sampled to 

11 initialize the segmentation at a finer resolution level, and refined by further 

12 segmentation at the finer resolution level. 

1 6. The method of claim 3, wherein assigning reference image pixels to 

2 motion classes comprises generating a separate motion class segmentation map 

3 for each pairing of the reference image and a respective neighboring image and 

4 merging the separate motion class segmentation maps into a unified motion class 

5 segmentation map for the reference image. 

1 7. The method of claim 6, wherein reference image pixels are 

2 respectively assigned to a motion class selected from a motion class set including 

3 a high motion class and a low motion class, and motion vectors assigned to the 

4 high motion class have higher magnitudes than motion vectors assigned to the 

5 low motion class. 

1 8. The method of claim 7, wherein merging the separate motion class 

2 segmentation maps comprises 

3 assigning a given reference image pixel to the low motion class in the 

4 unified motion class segmentation map when the given pixel is assigned to the 

5 low motion class in all of the separate motion class segmentation maps, and 

6 assigning a given reference image pixel to the high motion class in the 

7 unified motion class segmentation map when the given pixel is assigned to the 

8 high motion class in any of the separate motion class segmentation maps. 

1 9. The method of claim 8, wherein the motion class set further 

2 includes an intermediate motion class, and motion vectors assigned to the 

3 intermediate motion class have magnitudes higher than motion vectors assigned 

4 to the low motion class and lower than motion vectors assigned to the high 

5 motion class. 

1 10. The method of claim 9, wherein merging the separate motion class 

2 segmentation maps comprises assigning a given reference image pixel to the 

3 intermediate motion class in the unified motion class segmentation map when the 

4 given pixel is unassigned to the high motion class in any of the separate motion 
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5 class segmentation maps and is unassigned to the low motion class in all of the 

6 separate motion class segmentation maps. 

1 11. The method of claim 1, further comprising computing an alignment 

2 accuracy map for each pairing of the reference image and a respective 

3 neighboring image based on the computed motion maps. 

1 12. The method of claim 11, wherein computing alignment accuracy 

2 maps comprises re-mapping neighboring images to a coordinate frame of the 

3 reference image using respective motion maps, and computing correlation 

4 measures between pixels of the reference image and pixels of each of the motion- 

5 compensated neighboring images. 

1 13. The method of claim 11, further comprising up-projecting the 

2 computed alignment accuracy maps from the base image resolution level to the 

3 target image resolution level. 

1 14. The method of claim 13, further comprising up-projecting the 

2 motion maps from the base image resolution level to the target image resolution 

3 level, and classifying motion vectors in each up-projected motion map into valid 

4 and invalid motion vector classes based on the up-projected alignment accuracy 

5 maps. 

1 15. The method of claim 14, wherein values of target image pixels with 

2 corresponding pixels in all neighboring images being associated with motion 

3 vectors in the invalid motion vector class are computed by interpolating up- 

4 projected pixel values of the reference image. 

1 16. The method of claim 1, further comprising up-projecting the motion 

2 maps from the base image resolution level to the target image resolution level. 

1 17. The method of claim 16, further comprising re-mapping the 

2 neighboring images to the reference frame of the up-projected reference image 

3 using the respective up-projected motion maps. 
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1 18. The method of claim 1, wherein regions of the target image are 

2 respectively assigned to a motion class selected from a motion class set including 

3 a high motion class and a low motion class, and motion vectors of regions 

4 assigned to the high motion class have higher magnitudes than motion vectors of 

5 regions assigned to the low motion class. 

1 19. The method of claim 18, wherein computing target image pixel 

2 values in regions assigned to the high motion class comprises computing a pixel- 

3 wise combination of pixel value contributions from the up-projected reference 

4 image and the re-mapped neighboring images weighted based on pixel-wise 

5 measures of alignment accuracy between the reference image and the 

6 corresponding neighboring images. 

1 20. The method of claim 19, wherein pixel value contributions from the 

2 re-mapped neighboring images are additionally weighted based on measures of 

3 temporal distance between the reference image and the corresponding 

4 neighboring images. 

1 21. The method of claim 18, wherein computing target image pixel 

2 values in regions assigned to the low motion class comprises classifying low 

3 motion class reference image pixels and their corresponding pixels in the 

4 neighboring images based on measures of local texture richness. 

l 22. The method of claim 21, wherein low motion class reference image 



2 pixels and their corresponding pixels in the neighboring images are quantitatively 

3 evaluated for local texture richness, and are classified into a texture class selected 

4 from the texture class set including a high texture region class and a low texture 

5 region class, and pixels assigned to the high texture region class have higher local 

6 texture measures than pixels assigned to the low texture region class. 



1 23. The method of claim 22, wherein values of target image pixels 

2 classified into the low texture region class in the reference image and all of the 

3 respective neighboring images, are computed by interpolating up-projected pixel 

4 values of the reference image. 
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1 24. The method of claim 22, wherein a value of a given target image 

2 pixel classified into the high texture region class in the reference image or any 

3 respective neighboring images is computed based on a pixel value contribution 

4 from the up-projected reference image, and a pixel value contribution from a 

5 given re-mapped neighboring image weighted based on a measure of local texture 

6 richness computed for the given pixel, a measure of motion estimation accuracy 

7 computed for the given pixel , and a measure of temporal distance of the 

8 neighboring image from the reference image. 

1 25. The method of claim 18, wherein values of target image pixels are 

2 computed based on pixel value contributions from a number of base images 

3 neighboring the reference image, the number of neighboring base images being 

4 different for different motion classes. 

1 26. The method of claim 25, wherein values of target image pixels 

2 assigned to the high motion class are computed based on pixel value 

3 contributions from a fewer number of neighboring base images than the values of 

4 target image pixels assigned to the low motion class. 

1 27. The method of claim 26, wherein the motion class set further 

2 includes an intermediate motion class, the motion vectors of regions assigned to 

3 the intermediate motion class have lower magnitudes than motion vectors of 

4 regions assigned to the high motion class and higher magnitudes than motion 

5 vectors of regions assigned to the low motion class, and values of target image 

6 pixels assigned to the intermediate motion class are computed based on pixel 

7 value contributions from a fewer number of neighboring base images than the 

8 values of target image pixels assigned to the low motion class but a higher 

9 number of neighboring base images than the values of target image pixels 
10 assigned to the high motion class. 

1 28. An image processing machine, comprising at least one data 

2 processing module operable to: 

3 compute a respective motion map for each pairing of a reference image and 

4 a respective image neighboring the reference image in a sequence of base images, 
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5 each motion map comprising a set of motion vectors mapping reference image 

6 pixels to respective neighboring image pixels at sub-pixel accuracy; 

7 assign respective regions of a target image to motion classes based on the 

8 computed motion maps, the target image having a target resolution level and the 

9 base images having a base resolution level equal to or lower than the target 

10 resolution level; and 

11 compute pixel values for the target image based on corresponding pixel 

12 value contributions from the re-mapped base images selected in accordance with. 

13 the motion classes assigned to the target image regions. 

1 29. The machine of claim 28, wherein the at least one data processing 

2 module is operable to assign regions of the target image to motion classes by 

3 assigning pixels of the reference image to respective motion classes and up- 

4 projecting the motion class assignments to pixels of the target image. 

1 30. The machine of claim 29, wherein the at least one data processing 

2 module is operable to assign reference image pixels to motion classes by 

3 computing motion magnitude maps from each motion map, down-sampling the 

4 computed motion magnitude maps to a pyramid of motion magnitude maps at 

5 respective resolution levels lower than the base resolution level, and segmenting 

6 pixels in the pyramid of down-sampled motion magnitude maps into motion 

7 classes. 

1 31. The machine of claim 29, wherein the at least one data processing 

2 module is operable to assign reference image pixels to motion classes by 

3 generating a separate motion class segmentation map for each pairing of the 

4 reference image and a respective neighboring image and merging the separate 

5 motion class segmentation maps into a unified motion class segmentation map for 

6 the reference image. 

1 32. The machine of claim 28, wherein the at least one data processing 

2 module is operable to compute an alignment accuracy map for each pairing of the 

3 reference image and a respective neighboring image based on the computed 

4 motion maps. 
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1 33. The machine of claim 32, wherein the at least one data processing 

2 module is operable to compute alignment accuracy maps by re-mapped 

3 neighboring images to a coordinate frame of the reference image using respective 

4 motion maps, and computing correlation measures between pixels of the 

5 reference image and pixels of each of the motion-compensated neighboring 

6 images. 

1 34. The machine of claim 33, wherein the at least one data processing 

2 module is operable to up-project the computed alignment accuracy maps from the 

3 base image resolution level to the target image resolution level. 

1 35. The machine of claim 34, wherein the at least one data processing 

2 module is operable to up-project the motion maps from the base image resolution 

3 level to the target image resolution level, and classify motion vectors in each up- 

4 projected motion map into valid and invalid motion vector classes based on the 

5 respective up-projected alignment accuracy maps. 

1 36. The machine of claim 35, wherein values of target image pixels with 

2 corresponding pixels in all neighboring images being associated with motion 

3 vectors in the invalid motion vector class are computed by interpolating up- 

4 projected pixel values of the reference image. 

1 37. The machine of claim 28, wherein the at least one data processing 

2 module is operable to up-project the motion maps from the base image resolution 

3 level to the target image resolution level, and re-map the neighboring images to 

4 the reference frame of the up-projected reference image using the respective up- 

5 projected motion maps. 

1 38. The machine of claim 28, wherein: 

2 regions of the target image are respectively assigned to a motion class 

3 selected from a motion class set including a high motion class and a low motion 

4 class, and motion vectors assigned to the high motion class have higher 

5 magnitudes than motion vectors assigned to the low motion class; and 

6 at least one data processing module is operable to compute target image 

7 pixel values in regions assigned to the high motion class by computing a pixel- 
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8 wise combination of pixel value contributions from the up-projected reference 

9 image and the respective re-mapped neighboring images weighted based on pixel- 

10 wise measures of alignment accuracy between the reference image and the 

11 corresponding neighboring images. 

1 39. The machine of claim 38, wherein pixel value contributions from 

2 the re-mapped neighboring images are additionally weighted based on measures 

3 of temporal distance between the reference image and the corresponding 

4 neighboring images. 

1 40. The machine of claim 38, wherein the at least one data processing 

2 module is operable to compute target image pixel values in regions assigned to 

3 the low motion class based on classification of low motion class reference image 

4 pixels and corresponding pixels in the neighboring images based on measures of 

5 local texture richness. 

1 41. The machine of claim 28, wherein values of target image pixels are 

2 computed based on pixel value contributions from a number of re-mapped base 

3 images neighboring the reference image, the number of neighboring base images 

4 being different for different motion classes. 

1 42. A machine-readable medium storing machine-readable instructions 

2 for causing a machine to: 

3 compute a respective motion map for each pairing of a reference image and 

4 a respective image neighboring the reference image in a sequence of base images, 

5 each motion map comprising a set of motion vectors mapping reference image 

6 pixels to respective neighboring image pixels at sub-pixel accuracy; 

7 assign respective regions of a target image to motion classes based on the 

8 computed motion maps, the target image having a target resolution level and the 

9 base images having a base resolution level equal to or lower than the target 

10 resolution level; and 

11 compute pixel values for the target image based on corresponding pixel 

12 value contributions from the re-mapped base images selected in accordance with 

13 the motion classes assigned to the target image regions. 
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1 43. The machine-readable medium of claim 42, wherein the machine- 

2 readable instructions are operable to cause the machine to assign regions of the 

3 target image to motion classes by assigning pixels of the reference image to 

4 respective motion classes and up-projecting the motion class assignments to 

5 pixels of the target image. 

1 44. The machine-readable medium of claim 43, wherein the machine- 

2 readable instructions are operable to cause the machine to assign reference image 

3 pixels to motion classes by computing motion magnitude maps from each motion 

4 map, down-sampling the computed motion magnitude maps to a pyramid of 

5 motion magnitude maps at respective resolution levels lower than the base 

6 resolution level, and segmenting pixels in the pyramid of down-sampled motion 

7 magnitude maps into motion classes. 

l 45. The machine-readable medium of claim 43, wherein the machine- 



2 readable instructions are operable to cause the machine to assign reference image 

3 pixels to motion classes by generating a separate motion class segmentation map 

4 for each pairing of the reference image and a respective neighboring image and 

5 merging the separate motion class segmentation maps into a unified motion class 

6 segmentation map for the reference image. 



1 46. The machine-readable medium of claim 42, wherein the machine- 

2 readable instructions are operable to cause the machine to compute an alignment 

3 accuracy map for each pairing of the reference image and a respective 

4 neighboring image based on the computed motion maps. 

1 47. The machine-readable medium of claim 46, wherein the machine- 

2 readable instructions are operable to cause the machine to compute alignment 

3 accuracy maps by re-mapping neighboring images to a coordinate frame of the 

4 reference image using respective motion maps, and computing correlation 

5 measures between pixels of the reference image and pixels of each of the motion- 

6 compensated neighboring images. 

1 48. The machine-readable medium of claim 47, wherein the machine- 

2 readable instructions are operable to cause the machine to up-project the 
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3 computed alignment accuracy maps from the base image resolution level to the 

4 target image resolution level. 

1 49. The machine-readable medium of claim 48, wherein the machine- 

2 readable instructions are operable to cause the machine to up-project the motion 

3 maps from the base image resolution level to the target image resolution level, 

4 and classify motion vectors in each up-projected motion map into valid and 

5 invalid motion vector classes based on the respective up-projected alignment 

6 accuracy maps. 

1 50. The machine-readable medium of claim 49, wherein values of target 

2 image pixels with corresponding pixels in all neighboring images being associated 

3 with motion vectors in the invalid motion vector class are computed by 

4 interpolating up-projected pixel values of the reference image. 

1 51. The machine-readable medium of claim 42, wherein the machine- 

2 readable instructions are operable to cause the machine to up-project the motion 

3 maps from the base image resolution level to the target image resolution level, 

4 and re-map the neighboring images to the reference frame of the up-projected 

5 reference image using the respective up-projected motion maps. 

1 52. The machine-readable medium of claim 42, wherein: 

2 regions of the target image are respectively assigned to a motion class 

3 selected from a motion class set including a high motion class and a low motion 

4 class, and motion vectors assigned to the high motion class have higher 

5 magnitudes than motion vectors assigned to the low motion class; and 

6 the machine-readable instructions are operable to cause the machine to 

7 compute target image pixel values in regions assigned to the high motion class by 

8 computing a pixel-wise combination of pixel value contributions from the re- 

9 mapped neighboring images weighted based on pixel-wise measures of alignment 
10 accuracy between the reference image and the corresponding neighboring images. 

1 53. The machine-readable medium of claim 52, wherein pixel value 

2 contributions from the re-mapped neighboring images are additionally weighted 
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3 based on measures of temporal distance between the reference image and the 

4 corresponding neighboring images. 

1 54. The machine-readable medium of claim 52, wherein the machine- 

2 readable instructions are operable to cause the machine to compute target image 

3 pixel values in regions assigned to the low motion class based on classification of 

4 low motion class reference image pixels and corresponding pixels in the 

5 neighboring images based on measures of local texture richness. 

1 55. The machine-readable medium of claim 42, wherein values of target 

2 image pixels are computed based on pixel value contributions from a number of 

3 re-mapped base images neighboring the reference image, the number of re- 

4 mapped neighboring base images being different for different motion classes. 



